-<3)OH>3)Oa)>r  0Z03iHa>3 


AD-A274  313 


*»> 

Ji 


3 : 3 j I ^  1  zy  '  f  \ I >: ’  i  r>i>: i ^  ^ sT»s^ 


FOR  AIRCREW  TRAININQ  m  A  MULTISHIP  ENVIRONMENT 


WMtam  C.  Moor 


nt  of  IntfMlrW 
Coi<gt  of  ing 


EnginoorlnQ 


Tompo,  AZ  M2i7-BMS 


Do#  H.  AlKfrOWO 


HUMAN  RESOURCES  DIRECTORATE 
AIRCREW  TRAININQ  RESEARCH  DIVISION 
6001  South  Powor  Rood,  Building  558 
MOOO.AZ  85206-0904 


Novombor  1903 


DTIC 

ELECTE 
t*tC3  0  1993 


FInol  Tochnicol  Roport  tar  Poriod  July  1982  -  Octobor  1992 


Appravod  for  pubNc 


(MrttMition  is  unlimited. 


93-31410 

■iiuini 


93  l2  27  i24 


AIR  FORCE  MATERIEL  COMMAND 
BROOKS  AIR  FORCE  BASE,  TEXAS 


NOTICES 


This  technical  report  is  published  as  received  and  has  not  been  edited  by  the 
technical  editing  staff  of  the  Armstrong  Laboratory. 

When  Govemmert  drawings.  specWcations.  or  other  data  are  used  for  any  purpose 
other  than  in  connection  with  a  definitsiy  Oovemmert-reiated  procurement,  the  United 
States  Government  incurs  no  responstiifty  or  any  obigation  whatsoever.  Thetactthat 
the  Government  may  have  formulated  or  in  any  way  supplied  the  said  drawings, 
spedficatiorts,  or  other  data,  is  not  to  be  regarded  by  impHcssion,  or  otherwise  in  any 
manner  construed,  as  licensing  the  holder,  oranyother  person  or  corporation;  eras 
conveying  any  rights  or  permission  to  manufacture,  use,  or  seli  any  patted  invention 
that  may  in  any  way  be  related  thereto. 

The  Office  of  Public  Affairs  has  reviewed  this  report,  and  it  is  releasable  to  the 
National  Technicai  Information  Service,  where  it  wM  be  avalable  to  the  general  public, 
including  foreign  nationals. 

This  report  has  been  reviewed  arfo  is  approved  for  pubication. 


DEE  H.  ANDREWS 
Technicai  Director 


A 


^^ArCARROLL,  Colonel.  USAF 
ySnief,  Aircrew  1  raining  Research  Division 


REPORT  DOCUMENTATION  PAGE 


0704-0188 


n'llIJ  inUlii  I  ii^iiL'li«ii<i-«JH)»’iiitMjAi>«»<<iifcr-.Mi|iiiViili’lili  4rilli  #' Wti* Cjl^n  tii'i  'iiliJUw 


^pc?r 


1.  AGENCY  USE  ONLY  (Laave  blank)  2.  REPORT  DATE 

November  1993 


A  TITLE  AND  SUBTITLE 

Benefits  Estimation  for  Simulation  Systems  Used  for  Aircrew  Training  in  a 
Multiship  Environment 


6.  AUTHORfS) 
William  C.  Moor 
Dee  H.  Andrews 


X  REPORT  TYPE  AND  OATES  COVERED 
Final  -  July  1 992  -  October  1 992 


S.  FUNDING  NUMBERS 


7.  PERFORMING  ORGANIZATION  NAME(S)  AND  AOORESS(ES) 
Department  of  Industrial  and  Management  Systems  Engineering 
College  of  Engineering  and  Applied  Sciences 
Arizona  State  University 
Tempe,  AZ  85287-5906 


9.  SPONSORING/MONITORING  AGENCY  NAMES(S)  AND  ADDRESSfES) 
Armstrong  Laboratory  (AFMC) 

Human  Resources  Directorate 
Aircrew  Training  Research  Division 
6001  S.  Power  Road,  Bldg  558 
Mesa,  AZ  85206-0904 


11.  SUPPLEMENTARY  NOTES 

Armstrong  Laboratory  Technical  Monitor:  Dr  Dee  H.  Andrews,  (602)  988-6561 


9.  PERFORMING  ORGANIZATION 
REPORT  NUMBER 


10.  SPONSORING/MONITORING  AGENCY 
REPORT  NUMBER 

AUHR-TR-1 993-01 58 


12a.  OISTRIBUnON/AVAILABIUTY  STATEMENT 


12b.  DISTRIBUTION  CODE 


Approved  for  public  release;  distribution  is  unlimited. 


1XABSTRACT  (Maxunum  200  words) 

A  general  model  for  the  benefit  evaluation  of  multiship  training  simulation  systems  is  presented.  The  benefit 
measures  derived  are  oriented  toward  allowing  benefit-cost  evaluations  of  proposed  alternative  simulation 
systems.  The  model  is  based  on  empirical  data  drawn  from  evaluation  studies  of  simulators  as  well  as  analytical 
approaches.  The  focus  of  the  approach  using  this  model  is  selecting  the  most  appropriate  and  economic 
simulators  for  use  at  the  operational  level.  The  model  does  not  attempt  to  compare,  or  justify  the  comparison  of, 
the  training  value  of  a  specific  simulator  versus  a  specific  aircraft.  The  model  allows  for  full  sensitivify  analysis 
and  variation  of  all  important  parameters.  A  set  of  LOTUS  1-2-3  spreadsheets  are  presented  which  facilitate  the 
use  of  the  model.  The  model  is  demonstrated  by  means  of  an  application  which  is  based  on  actual  data. 


14.  SUBJECT  TERMS 
Aircrew  training 
Benefits 
Costs 


Flight  simulators 

Models 

Multship 


Simulation  systems 

Simulators 

Training 


1S.NUMBER  OF  PAGES 
48 


16.  PRICE  CODE 


17.  SECURITY  CLASSIFICATION 
OF  REPORT 

Unclassified 


N9M  794001 -ItO-SaOO 


It.  SECURITY  CLASSIFICATION 
OF  THI?  PAGE 

"ed 


19.  SECURITY  CLASSIFICATION  20.  UMITATION  OF  ABSTRACT 

UL 


Form  290  (Riv  2-00) 
Pmecnbod  by  ANSI  ^  230-10 
200-102 


CONTENTS 


Page 

INTRODUCTION .  1 

BACKGROUND .  2 

BASIS  FOR  BENEFITS  COMPUTATION .  6 

PERFORMANCE  AREAS .  9 

DETERMINATION  OF  NUSE  AND  CUSE  VALUES .  13 

DETERMINATION  OF  ESIM  VALUES  .  16 

COMPUTING  THE  NUMBER  OF  TRAINING  REPETITIONS  .  18 

COMPUTING  NUMBER  OF  SIMULATION  SORTIES .  19 

ESTIMATION  OF  BENEFIT  CONVERSION  FACTORS .  21 

BENEFITS  COMPUTATION  MODEL .  22 

TEST  OF  THE  BENEFITS  COMPUTATION  PROCEDURE .  22 

CONCLUSIONS  AND  RECOMMENDATIONS .  30 

REFERENCES .  36 


ETIC  QUALiTf  r. 


Accesion  for  y 

NTIS 

CRA^I  d 

OTIC 

•Ab 

'I 

-  bo  : 

- ....  _ 

_  _ _ 

- - - 

By 

Di  ,t 

■t  ■/ 

. .  ” 

L 

- 

Dist 

f\-l 

Sp. 

W.di 

_ 

Hi 


FIGURES 

Figure 

-M-Q.-  Page 

1  Flowchart  of  Benefits  Computation .  5 

2  isoutility  Curve  Illustrating  Pilot  Training 

Under  Current  Training  Conditions .  7 

3  Isoutility  Curve  Illustrating  Pilot  Training 

Under  "New”  Training  Jonditions .  8 

4  The  Air  Intercept  Trainer  (AIT)  .  28 


TABLES 

Table  Page 

No. 

1  Variable  Identification  and  Definition .  3 

2  F-15  Combat  Tasks  (Basis  for  Performance  Areas) .  10 

3  Selection  of  Performance  Areas 

Instructions  and  Measurement  Scale .  13 

4  Selection  of  Relative  Weights  of  Performance  Areas  .  14 

5  Determination  of  CUSE  and  NUSE  Values .  17 

6  Acquisition  of  ESiM  Values  . 18 

7  Data  Spreadsheet  (NUMCALC)  to  Compute  Number 

of  Simulation  Sorties .  20 

8  Spreadsheet  (NEWBEN.WK1 )  Showing  Method  for 

Computing  Benefits  .  23 

9  Performance  Areas  Selected  for  AIT  Evaluation .  29 

10  Relative  Time  To  Be  Spent  Training  Each 

Performance  Area .  30 

1 1  Total  Number  of  Simulation  Sorties  - 

AIT  Test  Case  .  31 

1 2  ESIM  Values  for  the  AIT  Example  Case .  32 

1 3  Benefits  Computation  Results  for  the  AIT  Test  Case .  33 


Iv 


PREFACE 


The  research  described  in  this  report  was  conducted  at  the  Armstrong 
Laboratory,  Human  Resources  Directorate,  Aircrew  Training  Research  Division 
(AL/HRA),  in  Mesa,  Arizona,  under  Work  Unit  1123*05-01,  In-House  Research 
and  Development  Support. 

In  1990,  Dr  William  C.  Moor  of  Arizona  State  University,  working  in  conjunction 
with  Dr  Dee  H.  Andrews  and  other  AL/HRA  personnel,  developed  a  preliminary 
model  for  the  benefit-cost  evaluation  of  multiship  simulator  alternatives.  This 
report  is  an  effort  to  improve  the  definitions  and  use  of  some  of  the  variables 
required  by  this  model. 


BENEFITS  ESTIMATION  FOR  SIMULATION  SYSTEMS  USED  FOR 
AIRCREW  TRAINING  IN  A  MULTISHIP  ENVIRONMENT 


INTRODUCTION 

Many  efforts  have  been  made  to  determine  the  "value”  of  aircraft  simulators 
for  training  military  pilots  (Barcus  &  Barcus,  1986;  Lethert,  1985;  and  Orlansky  & 
String,  1982).  While  none  of  these  could  be  deemed  a  failure,  neither  are  any 
regarded  as  wholly  successful  or  suitable  for  the  comparison  of  different  simulators 
intended  for  the  same  purpose. 

The  U.S.  Air  Force  (USAF)  desires,  insofar  as  possible,  that  proposed  capital 
expenditures  be  based  on  a  benefit-cost  comparison  among  all  competing 
alternatives  (Dept  of  the  Air  Force,  1988).  The  Aircrew  Training  Research  Division 
of  the  Armstrong  Laboratory  is  actively  engaged  in  research  on  the  development  of 
aircrew  training  simulators.  Some  simulators  (and  part-task  trainers)  have  been 
placed  with  operational  units  for  the  purposes  of  aircrew  research  and 
development.  A  difficulty  exists  in  that  no  widely  accepted  method  of  evaluating 
the  benefit-cost  impacts  of  these  devices  is  in  use.  Because  these  simulators 
represent  significant  capital  expenditures  (Marcus,  Patterson,  Bennett,  &  Gershan, 
1980;  Orlansky  &  Chatelier,  1983),  a  method  of  evaluating  their  benefit-cost 
relationships  is  desired  to  help  evaluate  their  usefulness  from  both  a  management 
and  a  research  perspective. 

Because  many  training  needs  exist  at  the  operational  (squadron  and  wing) 
level,  it  is  desired  that  simulators  be  evaluated  for  this  purpose  rather  than  strictly 
as  research  or  development  tools.  With  today's  technology,  it  is  possible  to  design 
a  simulation  system  that  can  represent  almost  all  tasks  a  pilot  might  be  called  on 
to  perform.  This  includes  some  tasks  that,  due  to  legal,  ethical,  safety,  or  security 
restrictions,  cannot  be  easily  practiced  in  the  aircraft  in  peacetime  even  though 
performance  of  the  tasks  would  be  expected  during  times  of  war.  In  addition, 
advances  in  communication  and  data  base  technology  makes  it  possible  to  link 
such  simulators  in  networks  enabling  many  pilots  to  engage  in  the  same  simulated 
exercises. 

The  above  factors  led  to  the  objective  of  developing  a  method  of  applying 
benefit-cost  analysis  to  simulators  which  are  designed  for  implementation  at  the 
operational  (squadron  or  wing)  level.  These  simulators  would  be  appropriate  for 
multiship  activity  and  training  (McDonald,  Broede,  &  Cutak,  1989).  The  purpose  of 
the  research  reported  in  this  report  Is  to  assist  in  accomplishment  of  this  objective. 
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BACKGROUND 


In  1990,  W.  C.  Moor,  working  in  conjunction  with  personnel  at  Armstrong 
Laboratory's  Aircrew  Training  Research  Division  (AL/HRA),  developed  a  preliminary 
model  for  the  benefit-cost  evaluation  of  multiship  simulator  alternatives  (Moor, 
1991a,  1991b;  Moor  &  Andrews,  1992).  This  model,  while  it  shows  promise  of 
meeting  the  objectives  stated  above,  is  in  need  of  refinement  and  application 
testing.  The  model  does  demonstrate  a  complete  method  of  benefit-cost  analysis 
of  multiship  simulation  alternatives  and  provides  a  means  of  computing  the  values 
for  this  analysis  in  a  manner  that  is  very  straightforward  (utilizing  LOTUS  1-2-3 
spreadsheets).  Because  much  of  the  work  presented  in  this  report  is  an  effort  to 
improve  the  definitions  and  use  of  some  of  the  variables  required  by  this  model, 
the  names  and  definitions  of  these  variables  are  shown  in  Table  1. 

The  original  model  developed  a  method  for  a  complete  benefit-cost  analysis. 
This  model  included  the  capacity  to  evalute  and  compare  multiple  simulation 
environments  as  an  explicit  element.  There  were  no  differences  in  the  computation 
method  based  on  simulation  environment.  Therefore,  this  study  focuses  on 
refining  the  method  of  benefit  determination  for  a  single  simulation  environment 
assuming  that  this  methhod  can  be  generalized  for  multiple  environments. 

The  current  research  focuses  on  the  benefit  component  because  it  is 
elements  of  the  benefits  computation  that  require  the  most  refinement.  The 
general  computational  model  for  benefits  determination  is  shown  in  Figure  1 . 

This  focus  on  the  benefits  component  is  supported  by  two  additional 
arguments.  The  first  of  these  is  that  the  original  cost  model  was  drawn  from 
established  Air  Force  policy  and  procedure  (Dept  of  the  Air  Force,  1988;  Knapp  & 
Orlansky,  1983).  Future  efforts  can  refine  the  cost  model  following  the  procedures 
originally  used  in  its  development.  It  is  not  anticiapted  that  the  overall  method  for 
benefits-cost  computation  and  comparison  would  be  significiatly  altered  by  these 
refinements. 

The  second  argument  is  that  several  operational  and  computational  issues 
with  respect  to  benefits  determination  had  not  been  resolved.  Chief  among  these 
is  the  use  of  the  shadow  cost  of  aircraft  use  as  a  basis  for  converting  benefits  into 
dollar  terms.  This  issue  is  addressed  in  the  next  section  of  this  report. 

The  overall  thrust  of  the  model  building  remains  the  same  as  for  the  original 
effort.  The  authors  desire  to  make  the  computation  model  as  clear  as  possible  to 
the  potential  user  and  to  build  it  in  a  form  that  facilitates  use.  In  this  case,  all 
computational  work  is  placed  in  LOTUS  1-2-3  spreadsheets  that  are  annotated  for 
data  entry  and  use.  The  model  is  built  in  reference  to  a  specif c,  operational 
airacraft  (chosen  by  the  analyst)  and  is  easily  modified  to  allow  comparisons  for 
any  air  superiority  jet  fighter  for  which  multiship  simulators  would  be  developed. 
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Table  1 

Variable  Identification  and  Definition 
(Extracted  from  Moor,  1991a  and  Moor.  1991b) 

Pwfonnanoe  Araa:  An  operational  activity  which  would  be  required  by  a  combat  pilot  and  would 
be  behaviorally  complex  enough  that  training  emphasizing  its  acquisition  and  maintenance  is 
appropriate.  The  Ee^ormance  Area  is  identified  as  PA(i)  where  i  refers  to  a  specific  performance 
area. 

ContiiHiation  Use  of  the  Simulator:  The  degree  to  which  a  simulator  would  be  used  to  train  in  a 
performance  area  after  initial  skill  training  had  been  accomplished.  The  Continuation  USE  is 
identified  as  CUSE(i). 

Necessity  of  Use  of  the  Simulator:  The  degree  to  which  a  simulator  must  be  used  to  train  in  a 
performance  area  (usually  because  of  extreme  hazard/danger  or  legality  of  operation).  The 
Necessity  of  USE  is  iden^ied  as  NUSE(i). 

Emulation  CapabKty  of  the  Simulation  Environnient:  The  degree  to  which  the  simulation 
environment  represents  the  actual  environment  experienced  in  the  aircraft  for  the  specific 
performarKe  area.  The  Emulation  capability  of  the  SIMulation  environment  is  identified  as  ESIMli). 

Simulation  Environment:  The  environment  (inside  the  simulator)  as  experienced  by  the  pilot.  The 
SIMulation  environment  is  identified  as  SIM(i);  where  j  refers  to  the  specific  simulation  environment 
(different  simulator). 

Aircraft  Training/ftactics  Sortie:  A  sortie  where  one,  or  more,  of  the  performance  areas  would  be 
practiced. 

1 .  Aircraft  Sortie  Duration  -  the  average  time  for  such  a  sor.ie.  The  Aircraft  sortie  HME 's 
identified  as  ATIME(i) 

2.  PerformarKe  Area  Iterations  -  the  number  of  times  the  specific  performance  area  could  be 
practiced  per  sortie.  The  Aircraft  REPetitions  are  identified  as  AREP(i). 

Simulation  Training/Practice  Sortie:  A  simulation  sortie  devoted  to  the  practice  of  one,  or  more, 
specific  performance  areas. 

1 .  Simulation  Sortie  Duration  -  the  average  time  for  such  a  sortie.  This  time  period  is  intended 
to  be  held  equal  to  the  corresponding  aircraft  sortie  duration  to  facilitate  later 
computations.  The  Simulation  sortie  TIMEs  are  identified  as  STIME(i). 

2.  Sinaiation  Performance  Area  Iterations  •  the  number  of  times  a  specific  performarKe  area 
could  be  practiced  per  simulation  sortie.  The  Sioiulation  REPetitions  are  identified  as 
SREPli). 
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Table  1  (Concluded) 


D^prae  of  Simulation  Compratsion:  Ratio  of  the  number  of  times  a  given  Performance  Area  can  be 
practiced  in  a  simulator  versus  an  aircraft.  The  Qegree  of  Simulation  Compression  is  identified  as 
DSC(i)  and  is  computed  by  SR(i)/AR(i) 

Simulation  Benefit  Factor:  This  factor  is  used  directly  in  computing  the  overall  benefits  imputed  to 
each  organizational  alternative.  The  Simulation  BENefit  factor  is  identified  as  SB^(i)  and  is 
computed  by  ESIM(i)*CUSE(i)*NUSE(i)*DSC(i|. 

Directly  Measured  Benefit  Bemarrts:  These  factors  are  based  on  the  shadow  costs  for  the  use  of 
aircraft  and  weaponry  approximated  by  the  marginal  costs  of  this  equipment. 

1 .  Marginal  (incramantai)  Aircraft  Cost  -  Cost  of  flying  the  aircraft  on  a  per  sortie  basis  (or  per 
hour.  SHAOAC$(i)),  corrected  for  Performarx:e  Area  if  appropriate.  The  Adarginal  Aircraft 
Cost  is  identified  as  MAC$(i). 

2.  Weaponry  Coat  -  Cost  of  using  ammunition,  weaponry  or  other  consumables  expended  per 
aircraft  sortie  for  each  Performance  Area.  This  cost  v^Hild  include  a  factor  for  all  damage 
(peacetime)  due  to  the  use  of  the  weaponry.  The  WEAPonrv  cost  is  identified  as 
WEAP$(i). 

IrKfrectly  Measured  Benefit  Elements:  These  factors  are  based  on  potential  losses  of  pilots  arxl 
aircraft  used  in  flying  sorties.  They  are  a  measure  of  risk  rather  than  training. 

1 .  Aircraft  Loss  Cost  •  Cost  of  loss  of  the  aircraft  as  a  function  of  its  use  in  flying  sorties  of 
the  specific  Performance  Area.  This  is  a  probability-based  measure  computed  by  (Cost  of 
an  aircraft.  TOTAC$|*(Probability  of  loss  per  sortie,  PLOSSAC).  The  AIRcraft  loss  Cost  is 
identified  as  AIRC$(i). 

2.  Pilot  Death  Cost  -  Cost  of  losing  a  pilot  due  to  training  accident  as  a  function  of  exposure 
to  risk  in  specific  Performance  Areas.  This  is  a  probability-based  measure  computed  by 
(cost  of  the  pilot,  TOTPIL$)*  (Probability  of  loss  per  sortie,  PLOSSPL).  The  PILot  death 
Cost  is  identified  as  PILCXi). 

Number  of  Simidation  Sorties:  The  total  number  of  simulation  sorties  that  can  be  performed  in  a 
specified  simulation  environment  for  each  PerformaiKe  Area  for  each  organizational  alternative  in  a 
year  (or  other  suitable  time  period).  The  NUMber  of  sorties  is  identified  as  NUM(i). 
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ASSUMPTION 
Av»rag«  Aircraft 
Sorti*  Time  (AT) 
sat  aqual  to 
Avaraga  Simulation 
Sortia  Tima  (ST) 
(ST/AT  -  1.00) 
Tharafora  not  usad 
in  this  mod  si 


Continuation 

Use 

(CUSE) 

h 

1 

1 

i 

Average  Aire 
RepetHions  ( 

If 

I  Average  Simulation 
I  Repetitions  (SR) 

I 

_ i 


Degree  of 
Simulation 
Compression 
(DSC  =  SR/AR) 


I 


I 


Number  of 
Simulation  Sorties 
Available  per  Year 
(NUM) 


Simulation 
Benefit  Factor 
(SBEN- 

ESIM*DSC*CUSE«NUSE) 


ESTIMATED  VALUES  FOR  BENEFIT  COMPUTATION 
(all  values  are  based  on  government  reports) 

Shadow  cost  of  an  aircraft  per  hour  (SHADACS) 

Cost  of  an  Aircraft  (TOTACS) 

Cost  of  trainiitg  a  pilot  (TOTPILS) 

Probability  of  aircraft  loss  per  flying  hour  (PLOSSAC) 
Probability  of  pilot  loss  per  flying  hour  (PLOSSPL) 


THE  ABOVE  ESTIMATES  ARE  USED  TO  COMPUTE 

THE  FOLLOWING  BENEFIT  FACTORS 

Cost  of  Aircraft  Cost  of  Aircraft 

Use  (MACS)  Damage/Loss  (AIRCS) 

Cost  of  Weapon  Cost  of  Pilot 

Use  (WEAPt) _ Injury/Loss  (PILCS) 


Computation  of  Benefits 

BEN  =  NUM  »  SBEN  «  (MACS  +  WEAPS  +  AIRC$  +  PILCS) 


Figure  1 

Flowchart  of  Benefits  Computation 

(Presented  in  terms  of  a  single  interface  and  a  single  performance  area) 

(Extracted  from  Moor  1 991  a) 
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BASIS  FOR  BENEFITS  COMPUTATION 


The  model  uses  the  shadow  price  of  aircraft  operation  as  the  basis  for 
benefits  computation.  It  has  been  argued  that  this  implies  a  direct  trade  of  aircraft 
hours  flown  in  exchange  for  hours  spent  in  the  simulator  using  the  two  variable 
economics  utility  trade-off  curve  (Orlansky  and  String,  1982).  This  trade-off  was 
never  intended,  implicitly  or  explicitly. 

While  there  are  no  perfect  metrics  for  the  creation  of  a  dollar  value  for 
benefits  imputed  to  a  project  (Maciariello,  1975;  McDonald,  et  al.,  1989;  Smith, 
1986),  Schmid  (1989)  provides  the  best  perspective  of  this  issue.  Arguably,  in 
Schmid's  discussion  of  the  methods  of  computing  benefit  values,  the  method  used 
here  would  appear  to  be  the  "Cost  Saving  Method"  (Schmid,  1989,  p.  66). 
However,  the  following  argument  shows  that  the  use  of  the  shadow  price  of  the 
aircraft  to  form  the  basis  of  dollar  estimation  of  benefits  is  more  appropriately  seen 
as  the  "Intermediate  Good  Method"  (Schmid,  1989,  p.  62)  and  this  estimate  is  a 
minimum  value  for  the  comparison  of  two  different  simulators. 

For  purposes  of  this  comparison,  utility  may  be  defined  as  the  degree  of 
"combat  readiness."  Figure  2  can  be  seen  as  presenting,  at  the  squadron  level,  this 
utility  in  terms  of  the  hours  of  training  received,  which  reflects  the  syllabus 
describing  the  mixture  of  tasks  which  have  been  trained.  The  iso-utility  curve 
shown  describes  the  "trade-off"  between  simulator  hours  of  the  simulators 
currently  in  use  and  aircraft  hours. 

Aircraft  hours  flown  is  defined  by  the  budgetary  process  which  establishes 
the  number  of  aircraft  hours  available  in  a  given  budget  period.  Then,  within  that 
budget,  the  number  of  hours  flown  may  be  reduced  and  replaced  by  the  current 
simulator  hours  but  cannot  be  increased.  The  replacement  of  aircraft  hours  with 
simulator  hours  would  demand  a  large  increase  in  total  training  hours  (in  order  to 
maintain  equivalent  training)  required  of  the  pilots  and  would  yield  no  better  trained 
pilot  than  the  current  situation. 

However,  for  a  given  budget  level,  it  is  reasonable  to  assume  that  the  best 
trained  pilots  possible  are  being  produced  (The  training  syllabus  is  as  good  as 
possible  in  its  mix  of  tasks  trained  and  flown,  for  that  budget  level.).  Therefore, 
there  is  no  good  reason  to  "trade-off"  aircraft  hours  for  simulator  hours.  At  a 
different  budget  level,  a  different  syllabus  and/or  mix  of  tasks  to  be  flown  would  be 
used  and  would  yield  a  different  level  of  utility  (training).  This  assumes  no  change 
in  the  simulators  being  used. 
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HOURS  IN  THE  AIRCRAFT 
PER  YEAR 


Figure  2 

Isoutility  Curve  Illustrating  Pilot  Training 
Under  Current  Training  Conditions 
(extracted  from  Moor,  1 992) 


If,  with  improved  models  of  simulators  and  for  a  gwen  level  of  aircraft  fhght 
hours,  an  apparent  increase  may  be  made  in  the  hours  of  simulator  time  available 
(and  corresponding  changes  to  the  training  syllabus  made)  the  isoutility  curve  is 
being  changed  (upward)  yielding  a  better  trained  pilot.  This  "new*  isoutility  curve 
is  one  that  is  defined  by  an  irKreased  number  of  aircraft  hours,  even  though  no 
additional  hours  are  authorized  under  the  budget.  This  is  illustrated  by  Figure  3 

Therefore,  using  the  current  budgeted  marginai  operating  cost  per  aircraft 
hour  as  a  starting  point  for  benefit  computation  (the  aircraft  shadow  price)  is 
justifiable.  Cost  per  aircraft  operating  hour  is  a  "savings*  (benefit)  at  the  next 


"haghor”  utilitv  curve  usocMted  with  the  improved  simuletors  lor  simuletor 
systatne)  that  created  the  curve.  Therefore,  this  baneht  is  s  ’fae”  measure  to  use 
to  compare  simulator  altematives  proposed  to  achieve  this  improvement  However 
due  to  the  nature  of  uamg  the  marginal  ctMt  as  a  best  estimate  of  the  shadow 
price,  if  a  different  utilitv  curve  actuaHy  applied,  the  cost  per  aircrafT  operatir^g  hour 
would  be  different  due  to  differerK:es  m  operating  econonrees  it  appears 
reasonable,  therefore,  to  stan  with  the  current  operating  cost  and  alter  it 
mcremantaNy  to  complete  a  parametric  or  sansitivitv  analysw 
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In  addition,  if  the  "new"  simulators  allowed  the  pilot  to  be  further  trained  in 
skills  which  are  currently  not  in  the  syllabus  or  which  can  be  trained  only  rarely  (for 
example,  air  combat  skills  currently  practiced  in  mator  exercises  available  to  the 
pilot  once  every  ye^  or  two),  then  even  further  changes  in  the  utility  curve  could 
be  argued.  The  benefit  cost  model,  based  on  the  shadow  price  of  aircraft  operating 
hours,  provides  a  way  to  choosa  between  competing  proposed  simulators  for  the 
provision  of  these  enhartced  skills. 

Therefore,  the  benefits  model  presented  in  this  paper  uses  the  budgetarily 
defined  number  of  aircraft  operating  hours  (doHars/hour)  as  the  basis  for  converting 
benefits  into  dollar  terms.  This  provides  a  benefit  comparison  value  for  different 
proposed  smti^tors  that  is  an  upper  limit  of  the  benefits  possible  for  each 
simulator 


PfRfOflMAMCf  AREAS 

Extensive  research  cortducted  using  the  F  15  simulators  at  McDonnell- 
Douglas  Aircraft  Company  (McAir)  (Houck  &  Thomas,  1989;  Houck,  Thomas,  & 

BeN.  1991 ).  has  yielded  an  empirically  derived  set  of  "tasks”  based  on  interviews 
with  pilots  that  form  the  current  basis  for  Performance  Areas.  A  list  of  the 
performance  areas  and  their  definitions  is  provided  in  Table  2. 

These  performance  areas  are  not  mutually  exclusive  and  therefore 
corrections  must  be  made  (allocations  denved)  when  computing  benefits.  This  is  a 
relatively  easy  correction  to  make  and  will  be  more  fully  presented  in  the  discussion 
of  determining  the  number  of  simulation  sorties  performed.  In  addition,  the 
performance  areas  do  not  form  an  exhaustive  set  of  all  pilot  behaviors,  however,  an 
allowance  is  made  to  add  performance  areas  to  this  list  as  necessary. 

These  performance  areas  have  the  advantage  of  being  meaningful  to  pilots 
who  can  compare  the  simulated  environment  to  the  environment  they  experience  in 
the  aircraft  This  yields  meaning  to  the  operational  values  for  the  simulator 
emulation  (ESIM)  variables.  In  addition,  these  areas  include  some  that  are  rarely 
erKOuntered  and  not  part  of  the  training  manual. 

The  set  of  performance  areas  is  probably  too  large  (27  areas)  to  grasp  for 
the  purposes  of  making  ESIM  comparisons.  It  is  proposed  that  when  a  simulator  is 
to  be  evaluated,  the  developers  of  the  simulator  select  those  areas  which  are  best 
presented  by  their  system.  The  number  created  would  probably  be  considerably 
smaller  and  would  be  most  advantageous  to  that  particular  simulator.  In  this  way, 
each  simulator  could  be  fairly  compared  on  its  own  attributes.  A  questionnaire  to 
facilitate  this  selection  has  been  developed  using  the  instructions  and  measurement 
scale  shown  in  Table  3. 
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Table  2 

F-1 5  Combat  Tasks 

A  Spanning  Set  of  Tasks  for  Muitiship  Operations 
Representative,  but  not  exhaustive 
Used  as  the  basis  for  the  identification  of  Performance  Areas 
(Extracted  from  Houck,  Thomas  and  Bell,  1 991 ) 


Taetica/Mbsian  Plannino  and  Briafina: 

V  a  a^^a  ^a  ^^a  aaa  aaa  aaa  a^aa  aa  ^aaa 

The  beginning  phases  of  the  flight.  Right  lead  does  specific  mission  planning  (e.g., 
weather,  target,  tactics,  threat,  etc.),  then  briefs  other  flight  noembers  concerning  the 
mission  plan. 


Maaion  fMafeHno: 

Postflight  discussion  of  how  closely  the  flight  adhered  to  the  briefed  game  plan,  reasons  for 
deviatiorts,  suggestkxos  for  improvement,  etc.  Should  be  used  as  a  laamirog  session. 

Escort  Tsedes: 

The  specific  tactics  to  be  used  for  escortirig  odoer  aircraft  (e.g.,  bombers,  electronic 
inteiiigerKe,  radar,  photo-reconnaissarrce),  to  protect  them  from  any  airborne  threat.  The 
aircraft  being  escorted  should  be  briefed  concerning  the  precise  mission  plan. 

VIsMl  Low  Loaal: 

Low  level  flight,  usually  flown  approximately  500  feet  above  ground,  using  visual 
raferances  for  positioning  and  turn  points. 

MghC  Tocdcs: 

Those  tactics  used  for  night  missions.  Usually  relies  more  on  radar  use  and  precisely 
briefed  tactics  artd  ntaneuvers  than  do  daylight  missions. 

Low  AMMude  Tactics: 

Tactics  specifically  designed  for  use  when  your  capability  to  "take  it  dovm”  is  limited  or 
nonexistent. 

VisiMl  Lockout: 

A  briefed  responsibility  of  each  flight  member  as  to  where  he  is  primarily  to  look  for 
threats.  For  a  single  ship  it  is  usually  expressed  as  a  percentage  of  time  available,  such  as 
70%  visual,  30%  radar. 


The  reverse  (percentage-wise)  of  visual  lookout, 
than  outside. 


More  time  is  spent  looking  at  the  radar 
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Table  2  (Continued) 


TKDCai  ronMnon: 

The  specific  place  each  wingman  should  fly,  with  respect  to  flight  lead,  and  his  role 
designed  to  accomplish  the  specific  mission,  considering  the  threat,  weather,  weapons, 
etc. 


Two-SHp  Tactics: 

Specific  tactics  designed  to  maximize  the  offensive  and  deferuive  capabilities  of  a  tv/o-ship 
flight. 

Four-Ship  Tactics: 

Specific  tactics  designed  to  maximize  dw  offensive  and  defensive  capabilities  of  a  four-ship 
flight. 

Bayood-Visual-Range  (BVR)  Employmant: 

Tactics  designed  to  operate  in  a  BVR  environment,  where  radar  and  radar  missile 
capabilities  must  be  considered. 

Al-Aspect  Defense: 

A  defense  based  upon  the  premise  that  the  enemy  has  the  ability  to  fire  weapons  from 
anywhere  in  a  360^  circle  around  the  fri«idly  aircraft,  as  opposed  to  a  guns-only 
environment,  where  the  enemy  must  fire  from  a  closenn,  stem  area. 

AB-Wssthsr  Eiwploymsnt: 

Employment  tactics  centered  around  radar  capabilities,  where  visual  weapons  may  not  be 
able  to  be  used. 

Comnraunications  Jamming: 

Tactics  designed  to  minimize  the  effect  of  enemy  communications  'amming. 

Tacticsl  Electronic  Wartaro  System  (TEWS)  Assessment: 

Use  of  the  onboard  TEWS  to  detect  potential  threats,  primarily  via  the  radar  warning 
receiver. 

Bectronic  Comtsrmsaiure/Electronic  Ownter-Countaimoasws  (ECM/E(XfM)  Emptoyment: 

Use  of  ECM  against  a  threat,  or  use  of  ECCM  against  enemy  ECM. 

Chaff /nare  Employmsnt: 

Use  of  ch^  to  defeat  enemy  radar  missiles  and  flares  to  defeat  enemy  infrared  missiles, 
based  upon  specific  tactics. 
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Table  2  (Concluded) 


n •action  to  Surfaoo-to^  Mhiiw  (SAMs): 

Maneuvers  designed  to  reduce  the  threat  from  or  to  defeat  SAMs. 

Reaction  to  Antiaireraft  Artiiery  (AAA): 

Maneuvers  arKl  tactics  desigr^  to  reduce  the  threat  from  ground  gunners. 

Reaction  to  Air  bitsrosptora  lAls): 

Maneuvers  and  tactics  designed  to  reduce  the  threat  from  ertemy  fighters. 

Radar  Bnployinsnt/Sorting: 

Tactics  used  for  radar  search  artd  the  sortirtg  of  enemy  formations  ar>d  individual  formation 
members. 

Visual  kfantification  (VIDI: 

Visually  determining  the  idemity  of  another  aircraft. 

Electrordc  Idantification  (BD): 

Using  electronic  systems  to  determine  the  identity  of  another  aircraft. 

Tactical  haaroaiit: 

An  intercept  using  specific  sirtgle-  or  multipie-ship  tactics,  using  cither  ground  control  radar 
or  ownship  radar. 

MuMbogay.  Four  or  More: 

Tactical  employment  against  multiple  enemy  air  threats. 

mifvni^K  wmniwVGHipnvi 

The  communications  used  between  flight  members,  usually  radio  #2  and  a  specific  discrete 
frequency. 


Even  in  the  case  of  two  simulators  which  provided  totally  different  training,  a 
comparison  utilizing  the  performance  areas  in  which  these  simulators  best  trained 
would  provide  an  indication  of  which  simulator  to  select,  if  only  one  could  be 
selected.  The  developers  of  the  simulator  provide  the  basis  for  comparison.  The 
benefits  computation  would  be  made  on  the  best  use  of  the  simulator,  operating  at 
its  maximum  efficiency.  Therefore,  selecting  the  one  with  the  best  benefit-cost 
ratio  or  maximum  benefit  minus  costs  should  be  easily  defended. 

Once  the  performance  areas  are  specified  by  the  developer  of  the  simulator, 
training  experts  from  the  USAF  would  be  asked  to  determine  the  relative 
desirability  of  each  of  these  areas  for  the  training.  No  performance  area  would  be 
eliminated  at  this  step,  but  a  relative  weighting  would  be  obtained  which  would  be 
used  to  determine  the  allocation  of  training  time  in  the  simulator.  This  allocation 
would  be  computed  by  multiplying  the  relative  weighting  (scaled  to  a  summed 
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Table  3 

Selection  of  PerforiTiance  Areas 
Instructions  and  Measurement  Scale 


Considering  each  of  the  following  air  combat  tasks,  please  evaluate  the  total  simulation 
anvirorMnant  (cockpit,  visual,  audio,  etc.)  in  terms  of  its  capabilitv  to  represent  the  task  from  the 
pilots  perspective  in  the  aircraft. 

Please  use  the  foUowirfg  scale  whan  evaluating  each  task. 

Unacceptable:  The  sinrujlator  is  totally  inappropriate  for  the  task, 

it  is  possible  that  negative  training  could  occur. 

Marginol:  Significam  deficieiKies  exist  which  reguir 

correction  before  widespread  use  of  the  sv  'ti- 

Adequate:  System  is  usable,  but  could/should  be  greatly 

improved. 

AocaptaMe:  Orriy  mirKK  deficierKies  are  rtoted. 

Fidhr  AecaptaMa:  No  improvemams  are  required. 

Qrde,  or  place  a  check  mark  on  the  evaluation  scale  for  each  task. 


vahM  of  1 .0)  by  the  number  of  hours  the  simulator  could  be  used  if  it  were 
operational.  No  simulator  sortie  would  ever  be  scheduled  to  train  only  one 
performance  area,  but  the  relative  weightings  are  assumed  to  be  constant  for  the 
purpose  of  computing  the  benefits  values.  A  questionnaire  has  been  developed  to 
facilitate  this  weighting.  The  instructions  and  measuring  scale  for  this 
questionnaire  are  shown  in  Table  4. 

DETEMMINATION  OF  NU8E  AND  CUSE  VALUES 

Once  the  performance  areas  are  selected  for  a  simulator  which  is  to  be 
evaluated  and  the  relative  training  err^hasis  for  each  of  these  areas  is  established, 
it  is  necessary  to  determine  the  utilization  characteristics  for  each  area.  Experts  in 
training  needs  for  the  USAF  would  be  asked  to  provide  an  evaluation  of  the  timing 
of  this  training  for  mission-ready  pilots.  This  evaluation  would  provide  a  measure 
of  the  continuity  of  training  need  (CUSE)  and  the  necessity  of  using  a  simulator  to 
meet  this  need  (NUSE). 


Table  4 

Selection  of  Relative  Weights  of  Performance  Areas 
Questionnaire  Instructions  and  Measuring  Scale 


EVALUATION  OF  THE  RELATIVE  TIME  SPENT  TRAINING 

Assuming  the  simulator  were  to  be  implemented  for  training  at  the  squadron  level  and  that  the 
followirH)  set  of  tasks  were  the  only  ones  to  be  trained  using  the  simulator. 

(List  of  performance  areas,  with  defirwbons,  derived  for  this  simulator) 

Show  the  relative  amount  of  time  that  should  be  spent  trainir>g  Task  A  versus  Task  B. 

Use  the  followir^  scale: 


1 .  Equal  amount  of  time 

2.  Barely  more  time 

3.  Weakly  more  time 

4.  Moderately  more  tinte 

5.  Definitely  more  time 

6.  Strongly  more  time 

7.  Very  strongly  more  time 

8.  Critically  nnora  time 

9.  Absolutely  more  time 


If  the  relative  time  should  be  reversed  between  any  two  tasks  indicate  by  showing  reversing 
arrows,  i.  e. 


< - 

TASK  A  TASK  B 

- > 


TASK  A  Relative  Time  TASK  B 


(Taaks  shown  on  a  paired  oonqtariaon  basis) 


The  values  for  CUSE  and  NUSE  are  both  set  to  1 .00  by  default,  unless  and 
until  there  is  a  clear  reason  to  set  them  to  some  other  value.  The  following 
decision  rules  have  been  developed  to  assign  these  values. 

CUSE  should  be  set  at  some  value  less  than  1 .00  when  it  is  known  that  a 
particular  simulator  is  only  useful  to  train  the  initial  phases  of  the  performance  area 
learning  curve.  CUSE  would  be  set  at  the  value  of  0.50  if  the  simulator  would  not 
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be  used  to  train  the  performance  area  at  the  mission-ready  squadron  level. 

Normally  this  would  occur  when  the  performance  area  was  routinely  practiced 
during  any  sortie. 

NUSE  should  be  set  at  a  value  of  2.00  if  the  performance  area  cannot  be 
trained  in  the  aircraft  at  the  squadron  level  (due  to  legal,  technological,  or  other 
restrictions)  although  the  aircraft  is  fully  capable  of  performing  its  portion  of  the 
task.  The  only  way  this  performance  area  may  be  trained  is  through  some  form  of 
special  exercise  or  in  a  non-flying  simulator.  NUSE  would  be  set  to  a  value  of  O.CX) 
if  the  performance  area  cannot,  and  should  not,  be  practiced  in  the  simulator. 

These  evaluations  are  made  with  respect  to  the  performance  areas,  not  with 
respect  to  the  specific  simulator.  The  questionnaire  asks  the  expert  to  rate  each 
performance  area  according  to  the  following  categories.  The  values  for  CUSE  and 
NUSE  are  shown  according  to  these  categories. 

1 .  CONTINUOUS  -  Performance  areas  which  are  continually  practiced  (or  ready 
to  use  on  an  intermittent  basis)  during  the  course  of  any  sortie.  No  sortie  is 
designed,  necessarily,  to  practice  these  performance  areas  but  they  are  performed 
(practiced)  as  needed,  e.g.,  Intraflight  Communications. 

There  would  be  no  multiplier  effect  for  these  performance  areas.  A 
simulation  sortie  to  practice  them  would  be  of  the  same  duration  and  nature  as  an 
aircraft  sortie  to  practice  them.  Generally,  specific  sorties  would  ofil  he  planned  to 
practice  these  tasks. 

CUSE  would  be  assigned  a  value  of  0.5,  NUSE  would  be  assigned  a  value  of 

1.0. 

2.  DISCRETE  CLASS  I  -  Performance  areas  which  are  performed  a  finite  number 
(N)  of  times  during  the  course  of  a  sortie  but,  logically,  would  never  be  performed 
more  than  N  times  during  any  sortie  of  approximately  equal  duration,  e.g..  Night 
Tactics,  Briefing/Debriefing. 

There  would  be  no  multiplier  effect  (number  of  repetitions)  involved  in  using 
the  simulator  to  practice  this  performance  area. 

CUSE  would  be  assigned  a  value  of  1 .0,  NUSE  would  be  assigned  a  value  of 

1.0. 

3.  DISCRETE  CLASS  II  -  Performance  areas  which  are  performed  a  finite 
number  (N)  of  times  during  the  course  of  a  sortie  planned  for  their  practice.  N 
repetitions  are  ail  that  can  be  performed  due  to  physical  constraints  on  the 
environment,  equipment,  and/or  pilot,  e.g..  Radar  Employment/Sorting. 
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There  would  be  a  multiplier  effect  if  the  simulator,  during  the  course  of  a 
sortie,  could  fully  repeat  the  task  N'  times.  N/N'  is  the  repetition  (REP)  correction 
to  apply  to  benefits  determination.  This  must  be  determined  by  analysts  familiar 
with  the  simulator  and  trainers  (or  subject  matter  experts  (SMEs))  familiar  with  the 
aircraft. 

CUSE  would  be  assigned  a  value  of  1 .0,  NUSE  would  be  assigned  a  value  of 

1.0. 

4.  DISCRETE  CLASS  III  -  These  are  performance  areas  which  can  be  performed 
during  war  or  under  extremely  unusual  and  rarely  occurring  conditions  (special 
exercises  or  locations  for  the  sortie).  Since  this  performance  area  is  designed  to  be 
performed  in  the  aircraft,  a  sortie  may  be  specified  and  planned  but  it  may  not  be 
actually  practiced  in  the  aircraft.  Therefore,  the  sortie  design  is  understood 
analytically. 

Strictly  speaking  there  is  no  multiplier  effect  accruing  due  to  the  use  of  the 
simulator  (since  N  =  0),  but  for  purposes  of  comparing  different  simulators,  N  may 
be  arbitrarily  set  equal  to  1 .  Then,  if  this  performance  area  may  be  practiced  one 
(or  more)  time  per  simulated  sortie,  the  benefit  factor  would  be  "corrected"  by  a 
factor  of  1  /I ,  any  further  practice  in  the  simulator  yields  a  REP  number  enhancing 
benefit. 

CUSE  would  be  assigned  a  value  of  1 .0,  NUSE  would  be  assigned  a  value  of 

2.0. 


These  definitions  and  numeric  assignments  of  values  have  been  developed 
during  the  course  of  this  report.  They  have  not  yet  been  validated  as  the  most 
appropriate  ones  for  the  purpose  of  computing  benefits  values. 

A  questionnaire  has  been  developed  to  ask  the  USAF  experts  how  each 
performance  area  could  be  trained  in  the  aircraft.  This  evaluation  is  used  to  provide 
the  basis  for  numeric  estimates  for  CUSE  and  NUSE  and  also  a  basis  for  the 
number  of  times  a  particular  performance  area  could  be  repeated  in  a  simulation 
sortie  versus  an  aircraft  sortie.  The  instructions  for  this  questionnaire  are  shown  in 
Table  5. 


DETERMINATION  OF  ESIM  VALUES 

An  empirical  test  of  a  questionnaire  developed  to  acquire  values  for  ESIM 
from  pilots  revealed  the  fact  that  all  pilots  questioned  (N  =  3)  wanted  to  reserve 
the  right  to  evaluate  the  capability  of  the  simulator  to  represent  the  performance 
area  in  two  dimensions.  One  of  these  was  the  dimension  of  initially  acquiring  the 
skills  necessary  to  perform.  The  second  dimension  was  the  maintenance  of  the 
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Table  5 

Determination  of  CUSE  and  NUSE  Values 
Questionnaire  Instructions 


TASK  CHARACTERISTICS 

The  follovving  are  a  set  of  descriptions  concerning  how  tasks  can  be  performed  (or  practiced) 
during  aircraft  sorties. 

Please  read  these  descriptions  and  apply  them  to  the  tasks  which  am  presented  on  the  next  page. 

1 .  CONTINUOUS  -  Tasks  which  are  continually  practiced  (or  ready  to  use  on  an  intermittent 
basis)  during  the  course  of  any  sortie.  No  sortie  is  designed,  necessarily,  to  practice  these  tasks 
but  they  are  performed  (practiced)  as  needed,  e.g..  Intraflight  Communications. 

2.  DISCRETE  CLASS  I  -  Tasks  which  are  performed  a  finite  number  (N)  of  times  during  the 
course  of  a  sortie  but,  logically,  would  never  be  performed  more  than  N  times  during  any  sortie  of 
approximately  equal  duration,  e.g..  Night  Tactics,  Briefing/Oebriefing. 

3.  DISCRETE  CLASS  II  •  Tasks  which  are  performed  a  finite  number  (N)  of  times  during  the 
course  of  a  sortie  planned  for  their  practice.  N  repetitions  are  all  that  can  be  performed  due  to 
physical  constrairrts  on  the  environment,  equipmem  and/or  pilot,  e.g.,  Rrirtg  a  particular  missile 
(there  are  only  N  missiles  per  aircraft). 

4.  DISCRETE  CLASS  III  -  These  are  tasks  which  can  be  performed  during  war  or  under 
extremely  unusual  arxl  rarely  occurring  conditions  (special  exercises  or  locations  for  the  sortie). 
Since  this  task  is  designed  to  be  performed  in  the  aircraft,  a  sortie  may  be  specified  and  planned 
but  it  may  not  be  actually  practiced  in  the  aircraft.  Therefore,  the  sortie  design  is  understood 
analytically. 


These  instructions  and  definitions  era  followed  by  a  listing  of  the  performance  areas  and  thair 
definitions  with  space  for  the  respondent  to  indkate  his  evaluation  of  the  characteristic. 


skills  necessary  to  perform.  Therefore,  two  questionnaires  were  developed  for  this 
measure.  The  exact  method  of  pooling  this  data  has  not  yet  been  determined 
(other  than  an  arithmetic  average).  Clearly  this  data  would  also  be  useful  as  a 
supplement  to  the  evaluation  of  the  CUSE  values. 

The  questionnaire  instructions  and’  measuring  scale  for  the  acquisition  of 
skills  are  shown  in  Table  6. 

The  instructions  and  measuring  scale  for  the  maintenance  questionnaire  are 
identical  to  the  acquisition  questionnaire  except  for  necessary  wording  changes 
("maintenance"  replaces  "acquisition"). 
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Table  6 

Acquisition  of  ESIM  Values 


EVALUATION  OF  SKILL  ACQUISITION  USING  THE  SIMULATOR 

Consider  each  of  the  following  Air  Combat  Tasks  and  using  the  following  scale,  rate  the  capability  of 
the  simulator  to  train  the  initial  acquisition  of  the  skills  necessary  to  perform  the  task  at  the  squadron 
level. 

Scale  of  Measurement 

Comparison  of  Learning  to  Perform  a  Task  in  the  Simulator  To  Learning  to  Perform  a  Task  in  the 

Aircraft. 


Measurement  Definition 

0.00  Absolutely  no  training/leamirtg 

*  potential  in  the  simulator.  The  task 

I  must  be  trained/leamed  entirely  in 

I  theF-15. 


5.00  The  task  can  be  partially  learned  in  the 

*  simulator  but  must  be  practiced  in  the 

I  F-1 5  to  be  fully  learned. 

I 

I 

* 

10.00  Perfect  training/leaming  errvironment  in  the 

simulator.  The  task  never  needs  to  be  practiced 
in  the  F-1 5.  Expectation  is  that,  the  first  time 
the  task  is  performed  in  the  F-1 5,  it  will  be 
performed  correctly. 


(This  is  foMowed  by  a  Rst  of  the  sslacted  perfonwanca  areas,  ttiair  definitions  and  space  for  the 
rsapondont  to  indkete  Ns/hor  numeric  evaluetion  of  that  arse.) 


COMPUTING  THE  NUMBER  OF  TRAINING  REPETITIONS 

One  of  the  major  advantages  that  a  simulator  enjoys  over  the  actual  aircraft 
is  that  a  given  task  may  be  practiced  repeatedly  without  the  need  of  refueling, 
rearming,  disengaging  and  reengaging  (unless  these  are  the  tasks  to  be  practiced) 
which  are  required  by  the  aircraft.  A  correction  factor  is  built  into  the  benefits 
computation  model  as  a  ratio  of  number  of  repetitions  possible  in  the  simulator 
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versus  the  number  of  repetitions  possible  in  the  aircraft  with  respect  to  a  given 
performance  area. 

There  would  be  no  correction  factor  for  any  performance  area  which  is 
classed  as  "Continuous"  in  its  nature  or  "Discrete  Category  I."  By  the  definitions 
of  these  terms,  the  performance  areas  so  described  would  always  be  practiced  the 
same  "number  of  times"  whether  in  the  aircraft  or  the  simulator. 

Performance  areas  which  are  identified  as  "Discrete  Category  11"  or  "Discrete 
Category  III"  would  be  those  which  permit  possible  increased  training  repetitions  in 
the  simulator.  Once  the  initial  categorization  of  performance  areas  is  accomplished, 
only  those  falling  into  the  latter  two  categories  would  continue  to  be  examined. 

Experts  in  air  training  sortie  design  would  be  polled  to  determine  how  many 
times  a  particular  performance  area  could  be  repeated  during  a  "normal"  training 
sortie  intended  for  its  practice.  Similarly,  experts  in  the  nature  of  the  simulator 
system  would  be  polled  as  to  how  many  times  this  performance  area  could  be 
repeated  in  a  simulation  sortie  under  correct  training  conditions. 

COMPUTING  NUMBER  OF  SIMULATION  SORTIES 

The  computation  of  benefits  for  a  particular  simulation  system  is  directly 
dependent  on  the  number  of  simulation  sorties  possible.  Obviously,  this  depends 
on  the  number  of  simulators  expected  to  be  in  use  and  the  operating  schedule  for 
that  use.  It  is  proposed  that,  when  simulators  are  to  be  compared,  operating 
conditions  as  nearly  equal  as  possible  be  used  (hours  of  operation,  number  of 
simulation  cockpits,  etc.).  The  only  differences  allowed  would  be  those 
technologically  intrinsic  to  the  simulator  (required  maintenance  downtime, 
reliability,  etc.). 

The  number  of  simulation  sorties  available  may  be  easily  computed  based  on 
the  organizational  configuration  being  examined  through  the  use  of  a  spreadsheet 
(identified  as  NUMCALC.WK1)  developed  for  this  project.  This  spreadsheet  and 
the  computational  equations  used  in  it  are  presented  in  Table  7. 

The  value  computed  by  this  spreadsheet  (NUM)  is  used  as  a  basis  for 
comparison  and  also  as  a  multiplier  for  the  relative  weightings  determined  for  each 
performance  area,  therefore  its  importance  cannot  be  overemphasized.  Every  effort 
must  be  made  that  completely  valid  figures  be  used  when  preparing  the  inputs  for 
the  computation  of  NUM. 
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Table  7 

Data  Spreadsheet  (NUMCALC)  to  Compute  Number  of  Simulation  Sorties 
(Shows  range  names  and  example  computational  equations) 
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ESTIMATION  OF  B0IEFTT  CONVENSION  FACTOfIS 

The  establishment  of  benefits  accrxjtng  to  any  simulation  alternative  is  based 
on  a  comparison  to  a  specific  aircraft  for  which  aircrew  training  is  necessary.  Most 
of  the  modeling  presented  to  this  point  (with  the  exception  of  several  specific 
references  to  sortie  training)  has  been  as  generic  as  possible.  This  would  facHitate 
the  evaluation  of  simulators  proposed  for  any  air  superiority  aircraft  training.  The 
final  conversion  of  modeled  variables  to  benefit  values,  however,  does  require  the 
specification  of  the  aircraft.  The  original  test  of  this  model  referenced  the  F-1 5  and 
this  reference  is  maintained  at  this  point. 

1 .  SHADOW  COST  OF  THE  AIRCRAFT  •  The  original  argument  for  the 
determination  of  the  shadow  cost  of  the  aircraft  was  that  the  appropriate  value  to 
start  with  was  the  marginal  operating  cost  of  flying.  This  argument  is  expanded  in 
an  earlier  section  of  this  report  utili2ir>g  the  concept  of  expanded  training  capability. 
Based  on  1 992/1 993  USAF  budget  figures  the  cost  is  estimated  (Dept,  of  the  Air 
Force,  1988a.  1991). 

SHADOW  COST  OF  THE  AIRCRAFT  =  $5,CXX)  per  flying  hour 

2.  WEAPONS  USE  BY  PERFORMANCE  AREA  •  The  model  includes  the  capacity 
to  treat  as  a  benefit  the  cost  savings  accruing  to  weapon  deployment  in  the 
simulator  that  does  not  represent  the  actual  consumption  of  the  weapon.  This 
value  depends  on  the  performance  area  being  trained;  many  performance  areas 
require  no  weapon  use,  others  require  a  variety  of  different  very  expensive 
weapons  to  be  used.  This  factor  is  in  the  rrK>del  arnl  is  available  for  further 
refinement  but  at  present  it  is  not  used. 

WEAPONS  COST  (PERFORMANCE  AREA)  =  $0 

3.  PILOT  COST  -  The  possibility  of  a  pilot  flying  a  sortie  incurring  an  accident 
which  leads  to  death  or  injury  is  very  real.  Training  in  a  non-flying  simulator  would 
reduce  this  possibility  to  negligible  terms.  Therefore,  a  benefit  is  computed 
corresponding  to  the  cost  of  training  the  pilot  multiplied  by  the  probability  of  death 
and/or  injury  per  flying  hour.  Currently,  this  benefit  factor  uses  only  the  cost  of 
training  a  pilot  (General  Accounting  Office,  1 987)  corrected  by  a  USAF  inflation 
correction  factor  (Directorate  of  Engineering  and  Services,  1988)  multiplied  by  a 
rough  estimate  of  pilot  death  while  flying  (Dept,  of  the  Air  Force,  1988).  The  cost 
of  injury  could  be  implemented  but  is  not  part  of  the  model. 

PILOT  COST  =  ($7,504,281  x  1.18028)  x  (0.0000205) 

=  $  1 82  per  flying  hour 
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4.  AIRCRAFT  COST  The  cost  accruing  due  to  loss  of  the  aircraft  is  treated  in 
a  similar  manner  and  for  a  similar  reason  as  the  cost  of  the  pilot.  The  benefit  factor 
allows  for  the  use  of  cost  of  damage  but  this  value  is  not  yet  implemented. 
Currently,  this  benefit  factor  uses  only  the  cost  of  replacing  the  aircraft  (drawn 
from  USAF  budget  figures  (Dept,  of  the  Air  Force,  1988a,  1988b)  multiplied  by  a 
rough  estimate  of  the  probability  of  the  total  loss  of  the  aircraft  (Dept,  of  the  Air 
Force,  1992).  Since  the  values  are  drawn  for  the  F-15  (not  F-15E),  this  must  be 
seen  as  an  arbitrary  estimate.  The  aircraft  is  no  longer  being  manufactured  and 
would  not  be  identically  replaced  if  lost. 

AIRCRAFT  COST  =  ($40,000,000)  x  (0.0000308) 

=  $1232  per  flying  hour 

BENEFITS  COMPUTATION  MODEL 

The  total  benefits  computation  model  that  was  originally  developed  has 
been  modified  to  include  all  the  factors  that  are  described  in  this  report.  The 
general  benefits  computation  equation  is  shown  in  Figure  1 .  In  addition,  editing 
comments  to  assist  in  data  entry  and  range  names  identifying  each  cell  have  been 
placed  on  the  spreadsheets  which  are  the  operational  representation  of  the  model. 
The  benefits  computation  spreadsheet  (NEWBEN.WK1)  is  described  in  this  section 
and  is  illustrated  in  Table  8.  The  spreadsheet  (NUMCALC.WK1)  to  determine  the 
number  of  sorties  (NUM)  operationally  available  for  any  specific  simulation 
configuration  has  been  described  in  a  previous  section.  The  files,  named  as 
indicated  and  in  LOTUS  1  -2-3  format,  are  available  to  anyone  who  wishes  to 
examine  them. 

The  benefits  spreadsheet  allows  for  up  to  eight  different  performance  areas 
to  be  named  and  entered.  More  performance  areas  may  be  used  but  the 
spreadsheet  must  be  modified  to  accommodate  this  increase  by  adding  rows  to  the 
spreadsheet  and  duplicating  the  computational  equations  as  necessary. 

The  spreadsheet  model  consists  of  a  number  of  input  and  output  vectors  and 
matrices  which  are  clearly  labeled  as  to  purpose.  It  includes  the  capability  of 
simultaneously  evaluating  up  to  four  distinctly  different  (or  four  variants)  simulator 
environments.  These  are  labeled  with  a  column  in  each  matrix  corresponding  to  a 
different  environment. 

TEST  OF  THE  BENEFITS  COMPUTATION  PROCEDURE 

A  preliminary  test  of  the  procedure  and  benefits  computation  model  was 
conducted  as  part  of  this  report.  The  simulator  used  as  a  focus  was  the  Air 
Intercept  Trainer  (AIT)  which  is  currently  in  use  with  several  Air  National  Guard  F- 
16  squadrons  and  which  was  developed  by  Armstrong  Laboratory  (Figure  4).  (It 
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Table  8 

Spreadsheet  (NEWBEN.WKi)  Showing  Method  for  Computing  Benefits 
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Table  8  (Continued) 
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Table  8  (Concluded) 

OVERALL  BENEFITS  IMPUTED  TO  THIS  ORGANIZATIONAL  ALTERNATIVE 
This  maVix  xltow*  vbIum  Ioi  all  compon*nt  parts  of  sach  of  tKa  simulation  intartscas. 
It  doas  NOT  show  tha  banafit  for  any  particular  orgarrizational  aKarnativa. 

Each  banafit  alamant  =  NUM(i.j)*SBEN(i.l')*{MACS(i)+WEAPt(i)-t^AIRC$(i)  +  PILC$(i)} 

This  is  tha  summation  of  all  alamants  of  tha  bartafit  mabix 
$121,012,603.31 

Sinca  thasa  Mluas  raprasant  tha  final  output  of  tha  computation, 
no  ranga  rwmas  ara  assignad  to  thasa  looatiorts 

12  3  4 

$1,250,438.40  $3,126,006.00  $1,045,126.40  $4,515,472.00 

$3,781,416.06  $4,726,771.20  $2,041,102.08  $6,827,558.40 

$5,672,125.44  $4,726,771.20  $2,041,102.08  $6,827,556.40 

$045,354.24  $4,726,771.20  $2,041,102.08  $6,827,558.40 

$625,210.20  $3,126,006.00  $2,017,660.60  $3,010,314.67 

$045,354.24  $4,726,771.20  $4,411,653.12  $4,551,705.60 

$1,800,706.48  $4,726,771.20  $4,411,653.12  $4,551,705.60 

$2,813,486.40  $4,680,144.00  $4,376,534.40  $4,515,472.00 


$17,024,103.36  $34,575,102.00  $26,885,062.88  $41,627,345.07 


should  be  noted  that  the  AIT  is  regarded  as  more  of  a  "multi-task"  part-task  trainer, 
than  as  a  full  multi-task  simulator.)  Several  of  the  AL/HRA  personnel,  who  had 
helped  develop  and  implement  the  AIT,  assisted  as  "experts"  for  questionnaire- 
based  input  and  three  pilots,  mission  ready  and  experienced  with  the  AIT  (although 
not  necessarily  F-16  qualified),  served  as  "experts"  for  pilot  input.  The  aircraft 
serving  as  the  basis  for  benefits  computation  was  the  F-16.  The  procedural  steps 
are  described  below. 

1 .  The  initial  step  was  to  administer  a  questionnaire  to  two  of  the  AL/HRA 
experts  asking  them  to  select  the  performance  areas,  from  the  list  of  27,  that  the 
AIT  was  designed  to  perform  and  their  impressions  of  the  quality  of  this 
performance.  Those  performance  areas  which  received  a  vote  of  "Acceptable"  or 
better  from  both  were  selected  as  the  evaluation  performance  areas.  These  yielded 
a  set  of  eight  performance  areas  as  a  basis  for  the  remaining  benefits  estimation. 
These  eight  areas  are  named  and  defined  in  Table  9. 


TOTAL 

ESTIMATED 

BENEFITS 


27 


Figure  4 

The  Air  Intercept  Trainer  (AIT) 
(Top  View) 


2.  The  same  two  experts  then  were  asked  to  perform  the  paired  comparison  of 
these  eight  areas  to  determine  the  relative  time  that  should  be  devoted  to  each  of 
them  in  allocating  simulation  use.  This  comparison  was  facilitated  through  the  use 
of  the  CRITERIUM  (Criterium  Reference  Guide,  1989)  software  package  which  uses 
a  comparison  scale  drawn  from  the  Analytic  Hierarchy  Process  (Saaty,  1980)  to 
automatically  determine  relative  weightings  (proportions)  and  consistency 
evaluations.  These  proportions  were  input  to  the  benefits  matrix  prior  to 
determining  the  number  of  simulation  sorties  that  could  be  flown  by  performance 
area.  These  proportions  are  shown  on  the  overall  benefits  estimated  model 
presented  in  Table  10. 

3.  The  decision  was  made  to  evaluate  one  pair  of  AIT  simulators  implemented 
into  one  Air  National  Guard  squadron.  The  AL/HRA  experts,  who  were  familiar 
with  the  operating  character  of  this  squadron,  provided  input  for  the  computation 
of  the  total  number  (NUM)  of  simulation  sorties  available.  The  results  of  this 
computation  are  shown  in  Table  1 1 . 

4.  A  different  AL/HRA  manager,  who  was  knowledgeable  about  the  AIT  and 
who  had  helped  develop  the  initial  list  of  performance  areas,  completed  the 
questionnaire  providing  preliminary  values  for  CUSE  and  NUSE.  These  values  are 
shown  on  Table  13.  In  this  table,  the  CUSE  and  NUSE  determinations  are 
highlighted  by  the  "C"  and  "DC2"  (the  only  two  categories  for  these  performance 
areas)  attached  to  the  names  of  each  performance  area. 


28 


Table  9 

Performance  Areas  Selected  for  AIT  Evaluation 


Radar  Lookout:  A  briefed  responsibility  of  each  flight  member  as  to  where  he  is  primarily  to  look 
for  threats.  For  a  single  ship  it  is  usually  expressed  as  a  percentage  of  time  available,  more  time  is 
spent  looking  at  the  radar  than  outside. 

Tacticai  Formation:  The  specific  place  each  wingman  should  fly,  with  respect  to  flight  lead,  and  his 
role  designed  to  accomplish  the  specific  mission,  considering  the  threat,  weather,  weapons,  etc. 

Two-Shk>  Tactics:  Specific  tactics  designed  to  mammize  the  offensive  and  defensive  capabilities  of 
a  two-ship  flight. 

Four-Ship  Tactics:  Specific  tactics  designed  to  maximize  the  offensive  and  defensive  capabilities  of 
a  four-ship  flight. 

Bayond-Visual-Range  (BVR)  Tactics:  Tactics  designed  to  operate  in  a  BVR  environrnem,  where 
radar  and  radar  missile  capabilities  must  be  considered. 

Radar  Search/Sorting:  Use  of  radar  for  search  and  sorting  of  enemy  formations. 

Tactical  kiteroapt:  An  intercept  using  specific  sirtgle  or  multiple  ship  tactics,  using  either  ground 
control  radar  or  ownship  radar. 

Midtibogey,  Two  or  Mora:  Tactical  employment  aganst  multiple  enemy  air  threats. 


5.  The  three  pilots  then  evaluated  each  of  the  eight  areas  with  respect  to  the 
capability  of  the  AIT  to  provide  emulation  capability  with  respect  to  the  aircraft.  (It 
was  at  this  point  that  the  distinction  between  "acquisition  of  ability"  and 
"maintenance  of  ability"  was  requested  and  accepted  for  current  purposes.)  The 
averages  for  each  performance  area  and  each  use  was  maintained  yielding  a  total 
of  two  simulation  environments  to  be  compared.  The  data  for  ESIM  evaluation  is 
presented  in  Table  12,  average  values  are  used  for  benefits  computation. 

6.  All  experts  were  polled  to  determine  the  expected  number  of  repetitions  of 
each  performance  area.  This  same  poll  yielded  average  duration  of  aircraft  sortie 
and  simulation  sortie.  This  information  was  also  used  to  complete  the  numerical 
evaluation  of  the  CUSE  and  NUSE  variables. 

7.  All  benefits  computation  values  presented  earlier  (in  the  context  of  the  F-1 5) 
were  corrected  to  be  representative  of  the  F-1 6. 

The  final  results  of  this  preliminary  test  are  shown  in  Table  1 3.  Due  to  the 
preliminary  nature  of  the  data  collection  instruments  this  should  not  be  considered 


29 


Table  10 

Relative  Time  To  Be  Spent  Training  Each  Performance  Area 

AIT  Example 


RELATIVE  PROPORTION  OF  TIME  SPENT 
EVALUATORS  RATINGS 


NAME  OF  PERFORMANCE  AREA 

Beyond-Visuai-Range  Emptoyment 

Radar  Lookout 

Tactical  Formation 

Two-Ship  Tactics 

Four-Ship  Tactics 

Radar  Empioyment/Sortir>a 

Tactical  Intercept 

Multibogey,  Four  or  More 


1  2  AVERAGE 


24.87% 

15.27% 

20.07% 

14.08% 

38.64% 

26.36% 

3.02% 

7.62% 

5.32% 

8.22% 

3.90% 

6.06% 

2.99% 

2.27% 

2.63% 

25.15% 

20.55% 

22.85% 

18.60% 

9.66% 

14.13% 

3.07% 

2.10% 

2.59% 

a  true  benefits  evaluation  of  the  AIT.  It  would  be  extremely  misleading  to  interpret 
these  numerical  values  as  anything  other  than  tentative  at  best. 

CONCLUSIONS  AND  RECOMMENDATIONS 

This  report  improved  and  refined  the  benefits  model  developed  in  previous 
work.  It  created  the  operational  procedures  necessary  to  acquire  all  data  required 
for  estimating  benefits.  In  addition,  the  report  completed  an  operational  test  of 
these  procedures  demonstrating  their  feasibility. 

The  report  did  not  address  additional  refinement  of  the  original  cost  model  or 
the  model  used  to  present  benefit  cost  summary  information.  These  models  are 
available  as  originally  developed  in  1 990. 

Obvious  areas  requiring  future  research  are  listed  in  the  report.  These 
include  improvements  and  refinements  in  the  methods  of:  a)  acquiring  and  using 
the  CUSE  and  NUSE  values;  b)  validating  the  ESIM  values;  c)  validating  the  use  of 
the  master  list  of  performance  areas;  d)  justifying  the  proportionality  values  for  the 
number  of  simulation  sorties  per  performance  area;  and,  e)  determining  the  general 
usability  of  the  operational  procedures.  Additional  areas  for  further  development 
are  those  cited  above. 
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Table  1 1 

Total  Number  of  Simulation  Sorties  -  AIT  Test  Case 
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NOT  USED  FOR  THIS  EXAMPLE 


Table  13 

Benefits  Computation  Results  for  the  AIT  Test  Case 
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Sum  of  propcxtions 
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